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(54) Abstract Title 

Establishing a secure environment for a software process 

(57) A system and method for establishing a secure execution environment for a software process analyses 
system application program interface (API) calls to determine whether the system API call executes a new 
program. If the system API call executes a new program, the system and method analyse the call to determine 
whether an execution environment, in the form of a plurality of attributes is attached to an executable program 
file associated with the new program. An execution environment contains security attributes that are to be 
associated with the resulting process invoked by execution of the new program. If an execution environment 
is attached, the invention assigns the attributed of the execution environment to the new process, thereby 
ensuring that the new process includes the security features and capabilities specified in the execution 
environment. 
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SYSTEM AND METHOD FOR ESTABLISHING A SECURE 
ENVIRONMENT FOR A SOFTWARE PROCESS 

The present invention relates generally to computer systems, and, more 
particularly, to a system and method for establishing a secure execution environment for a 
software process executed by a computer program. 

With the advent of, and ever increasing access to, the Internet, computer system 
security has become and continues to gain in importance. Typically, information to be 
accessed via the Internet (for example, but not limited to, a web site having one or more 
web pages) resides on a computer system, which is connected to the Internet. This 
computer system is referred to as a "web server." The computer system maintains the 
Internet information in a memory and executes what is referred to as a "web server 
program." When the web server program is executed, a process corresponding to the 
web server program is established on the computer. In this manner, those individuals 
wishing to access the web site over the Internet initiate, from their own computer, a web 
browser program, connect to the Internet, enter the required uniform resource locator 
(URL) associated with the web site sought, and then view the web pages corresponding 
to the web site. 
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Much information contained on web servers and exchanged over the Internet is 
confidential in nature. For example, a merchant operating a web site would like to 
maintain as confidential the credit card information that is received from customers over 
the Internet and stored on the merchant's web server. 

Unfortunately, due to the nature of available web server programs, it can be 
relatively easy for an unauthorized individual to access a web server through an Internet 
web site and gain access to confidential information. Furthermore, all programs that 
execute on a computer have corresponding processes associated therewith. For 
example, each time a word processing program is initiated on a computer, a 
corresponding process executes within the computer so thatthe word processor program 
■nay operate. Indeed, each time any computer program is initiated, a corresponding 
process is launched to enable the program to run. Each process runs in a unique, 
protected domain encompassing its program code, data and other resources. Each of 
these processes communicate in what is called the "user mode" through an application 
program interface (API), with what is known as a "kernel." Each process initiates 
^system calls;' through which the process communicates with the kernel in what is 
referred to as "kernel mode." When operating in keme, mode, the kernel utilizes certain 
protected functions of the computer processor (not available to user mode programs) in 
orderto execute functions requested by the process. 

Part of the functionality of the kernel is to assign and maintain a list of attributes 
for each process. Among other uses, the attributes are used by the kernel to perform an 
access control function, so that, for example, a particular process while executing on 
behalf of a particular user has access privileges assigned that may be different from that 
of processes operating on behalf of other users. In many cases, a number of different 
programs execute in processes that have been assigned identical sets of attributes. In 



these cases, the kernel is unable to apply different access control criteria to the different 
processes, and a process may thus be vulnerable to various flaws that can cause it to 
corrupt data or resources belonging to another process. 

The present invention seeks to provide a system and method for creating a secure 
execution environment in which a program may execute. 

According to an aspect of the present invention there is provided a system for 
establishing a secure execution environment for a software process executed by a program 
operating on a computer, comprising a software process including a plurality of attributes 
operating on a computer, an operating system kernel in communication with the software 
process and in communication with an executable file to be accessed by the software 
process, and a system call trap associated with the operating system kernel, the system call 
trap configured to assign a selected plurality of the attributes to the software process, the 
selected plurality of attributes stored in association with the executable file. 

According to another aspect of the present invention there is provided a method for 
establishing a secure execution environment for a software process executed by a program 
operating on a computer, the method comprising the steps of: operating a software process 
including a plurality of attributes on a computer, executing an operating system kernel in 
communication with the software process, the operating system kernel in communication 
with an executable file to be accessed by the software process, and assigning a selected 
plurality of the attributes to the software process, the selected plurality of attributes stored 
in association with the executable file. 

The preferred embodiment can secure the contents of a program/process, such as a 
web server program, by compartmentalizing the process in which each program operates 
and to control the capabilities granted to the process by dynamically assigning a unique 
differentiable set of attributes to it. 

An embodiment of the present invention in described below, by way of example 
only, with reference to the accompanying drawings, in which: 

FIG. 1 is a schematic view illustrating a preferred embodiment of execution 
environment system; 

FIG. 2 is a block diagram illustrating, in further detail, the kernel of FIG. 1; 
FIG. 3 is a block diagram illustrating, in further detail, the system call trap of 

FIG. 1; 

FIG. 4 is a block diagram illustrating the kernel of FIG. 1 and the effect of a 
"createprocess" system API call; 



3 



execac Wlto , . dlscreIe , ^ Qf 

fcve access co^o, a* ^ ^ ^ JJ^™ *" 



4 



single computer environment, the system and method for creating a secure execution 
environment can be implemented in a networked computing arrangement in which a 
number of computing devices communicate over a local area network (LAN) or over a 
wide area network (WAN). 

The system and method for creating a secure execution environment can be 

implemented in hardware, software, firmware, or a combination thereof. The 
preferred embodiments are implemented in software or firmware that is 

stored in a memory and that is executed by a suitable instruction execution system. If 

implemented in hardware, as in an alternative embodiment it can be 
implemented with any or a combination of the following technologies, which are all 
well known in the art: a discrete logic circuits) having logic gates for implementing 
logic functions upon data signals, an application specific integrated circuit (ASIC) 
having appropriate combinational logic gates, a programmable gate array(s) (PGA), a 
field programmable gate array (FPGA), etc. 

The secure execution environment program, which comprises an ordered 
listing of executable instructions for implementing logical functions, can be embodied 
in any computer-readable medium for use by or in connection with an instruction 
execution system, apparatus, or device, such as a computer-based system, processor- 
containing system, or other system that can fetch the instructions from the instruction 
execution system, apparatus, or device and execute the instructions, hi the context of 
this document, a "computer-readable medium" can be any means that can contain, 
store, communicate, propagate, or transport the program for use by or in connection 
with the instruction execution system, apparatus, or device. The computer readable 
medium can be, for example but not limited to, an electronic, magnetic, optical, 
electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation 
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As shown in FIG. 1, computing device 101 also executes web server 1 1 1, which 
when launched, invokes process B 108. Similar to that described above with respect to 
process A 106. process B 108 is also associated with corresponding attributes 1 14. 
which are also maintained by the kernel 127, and which may include credentials 1 1 8, 
such as user ID, group IDs and privileges. The attributes fall into three general 
categories as follows. 

1 ) Identifiers: identify either the particular user on whose behalf the process is 
executing, or one of a number of different groups to which a user might belong (on 
which various access control decisions might be based: i.e. you can only execute this 
program if you belong to the "administrators" group). 

2) Privileges: one of a number of discrete individually assignable tokens (to be 
described below) that can be granted to a process. These are typically used to grant a 
special capability. For example, on the Windows NT operating system, only if a 
process holds the "SeSystemtimePrivilegc" privilege can it set the computer's notion 
of the time, or only if the process holds the "SeAuditPrivilege", can one configure, 
enable or disable auditing on the system. Similarly, only if the process holds the 
"SeShutdownPrivilege", can the operating system be instructed to shut the system 
down. 

3) Other attributes: For example, on the Windows NT operating system, this would 
include assigning the "default owner" ED to any newly created file, or assigning the 
default access control list to a newly created file. 

Furthermore, although illustrated using only two processes, the execution 

environment system 100 is capable of supporting many additional 

application programs and their corresponding processes, such as, for example but not 

limited to, a file transfer process, a mail server process, etc. Further still, it is 



contemplated that an application program may have more than one process running 
sinuiltaneously. 

Computing device 101 also includes system call application program interface 
(API) 1 20. The functionality described above with regard to process A 1 06 and process 
B 1 08 occur at the application level, or what is referred to as the "user mode." The 
functionality to be described below, and on the opposite side of API 120 from the user 
mode, occurs in what is known as the operating system level, also referred to as the 
•kernel mode." For example, process A 106 communicates with kernel 127 through 
system call trap 300 via connection 121, and process B 108 communicates with kernel 
127 through system call trap 300 via connection 122. The communication between the 
processes .06 and 1 OS, and kernel 127 occur using API 120 through the use of what are 
known as -.system calls." The system calls allow those processes operating in the user 
•node to communicate via the API 120 with those elements operating in the kernel 
mode. 

Kerne! ! 27 includes process attributes 200, which correspond to the attributes 
H2a»d ! 14 described above. Kernel 127 communicates via connection 126 with 
processor 124 and communicates with file 136 via connection 134. Illustratively, file 
1 36 is a file that is accessed by, for example, process A 1 06 or process B 1 08. For 
example, file 136 can be an executable program file, such as a word processing 
document that word processor 109 desires to access. Typically, there are many 
executable files 136 maintained in a storage device (not shown) residing within 
computing device 101. File 136 is representative of one or more executable files 
associated with any application program that might be running on computing de 



levice 



101. Each instance of file 136 also includes an associated unique access control list 138, 



shown via connection 137. Access control list 138 includes information relating 



to users 



and the users" access to the particular file 136. File 136 is also associated with 
execution environment attributes 225 via connection 13 1 . In accordance with an aspect 
of the invention, the execution environment attributes 225 are associated with each 
stored executable program file 136, and can be used to define the contents of the 
attributes assigned to newly created processes to be explained below. 

In the preferred embodiment, system call trap 300 (to be 
described in further detail below) intercepts system API calls destined for kernel 127 
from process A 1 06 and process B 108. System call trap 300, in cooperation with kernel 
127 uses the execution environment attributes 225 to develop the process attributes 200 
within kernel 127, in order to assign the appropriate attributes to processes 106 and 108. 

As mentioned above, each program executing on computing device 101 includes 
an associated process, such as process A 106 associated with word processor 109 and 
process B 1 08 associated with web server 111. The kernel 1 27 maintains a process table 
with one entry for each process. A process table entry contains the process attributes 
200 associated with each process. In operation, the word processor 109 and the web 
server 1 1 I request services (such as opening a file, reading a directory, allocating 
memory, etc.) from the kernel 127 by issuing a system API call via connections 121 and 
122, respectively, via API 120. The kernel 127 responds to those system API calls in 
accordance with the process attributes 200. 

FIG. 2 is a block diagram illustrating, in further detail, the kernel 127 of FIG. I. 
As shown in FIG. 2, kernel 127 contains process A attributes 201, which correspond to 
process A 106, and contains process B attributes 202 which correspond to process B 
108. The process A attributes 201 contained within kernel 1 27 correspond to the 
attributes 1 12 of FIG. 1 , and the process B attributes 202 correspond to the attributes 
11 4 of FIG. 1 . Kernel 127 also includes an access token A 206 corresponding to process 
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A attributes 201 and process A 106, and includes access token B 205, which 
corresponds to process B attributes 202 and process B 108. 

For example, access token B 205 includes the security attributes associated with 
process B 1 08 (including the credentials 1 1 8 (FIG. 1) associated with the user under 
whose logon the process is executing). As mentioned above, credentials 1 16 associated 
with process A 1 06 and credentials 1 18 associated with process B 108 can include the 
user ID, group IDs and a list of privileges (special user rights). 

FIG. 3 is a block diagram illustrating, in further detail, the system call trap 300 
of FIG. 1 . As shown in FIG. 3, system call trap 300 resides at the operating system level 
(kernel mode), logically located between kernel 1 27 and the processes (A 1 06 and B 
1 08) that execute on computing device 101 . The system call trap 300 intercepts system 
API calls sent through API 120 before they reach the kernel 127. The system call trap 
300 manipulates the system API call parameters prior to forwarding the calls to the 
kernel 127. In the preferred embodiment, the system call trap 300 may 
modify the contents of access tokens, corresponding to the processes, as the processes 
and access tokens are created. For example, system call trap 300 includes process A 
attribute extension 301 and process B attribute extension 302. Process B attribute 
extension 302 corresponds to process B attributes 202 and process B 108. Process B 
attribute extension 302 conesponds to process B attribute 202 via connection 311. 

Furthermore, process A attribute extension 301 is associated with a 
corresponding access token A extension 304, and process B attribute extension 302 is 
associated with a corresponding access token B extension 305. Access token B 
extension 305 conesponds to access token B 205 via connection 312. Access token B 
205 may inch.de modified attributes 315, and access token B extension 305 may include 
additional attributes 310, which are logically added to process B 108 through the 
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operation of the invention. The creation of the modified attributes 315 and the 
additional attributes 310 will be explained in detail below. 

Process B attribute extension 302 is maintained in what is referred to as a 
"shadow table" linked via connection 3 1 1 to process B attributes 202. Similarly, access 
token B extension 305 is maintained in a shadow table that is linked via connection 3 1 2 
to access token B 205. In this manner, system call trap 300 can intercept API calls 
between processes executing in the user mode and kernel 127, and can add attributes to 
each executing process. The additional attributes 3 10 in access token B extension 305 
and the modi (led attributes 3 1 5 in access token B 205 are obtained from the execution 
environment attributes 225 (FIG. 1) that will have been previously stored as an 
attachment to the stored executable program file 136 (FIG. 1). 

FIG. 4 is a block diagram illustrating the kernel 127 of FIG. 1 and the effect of a 
"createprocess" system API call on tKekernel 127. In the example shown in FIG. 4, 
process A I06ls'executing and seridsji^createprocess" system API call via connection 
121 through API 120. In this example, process A attributes 201 and access token A 206 
are already in place. During initial system start, or "boot-up", a portion code executes in 
the operating system that creates the very first process. Subsequently, a new process is 
always created by an existing process through execution of a createprocess system API 
call as mentioned above. Continuing with the example, when the createprocess API call 
is sent, process B attributes 202 are invoked, and a copy of access token A 210 is created 
from access token A 206. In this manner, the contents of access token A 206 are 
automatically assigned to the process B attributes 202 in the form of a copy of access 
token A 2 1 0. hi this example, the attributes assigned to process B 1 08 are the same as 
those assigned to process A 106. 
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associated with a stored executable program file (/.,, execution environment attributes 
225 associated with file 136 of FIG. 1), such that when the program file is executed, the 
new process's access token is modified as specified by the information in the execution 
environment. The additional attributes may augment, reduce or completely replace the 
corresponding attributes (that would ordinarily be copied from the first process's access 
token). The ability to dynamically assign these attributes allows a heretofore 
unrecognized level of control over process security attributes. Each execution 
environment contains a flag indicating whether the attributes specified are also to be 
applied to processes subsequently created by the affected process, or whether 
subsequently created processes should revert to the original set of attributes. This flag is 
referred to as die ''iiil^ritableflag.'' 

Referring now to FIG. 5, assuming that process A 106 is executing, a 
createproccss system API call is communicated via connection 121 through API 1 20 to 
kernel 127, and intercepted by system call trap 300. Process A attributes 201 and access 
token A 206, within kernel 127, already exist because process A 106 is currently 
executing. Similarly, process A attribute extension 301 and access token A extension 
304, within system call trap 300, already exist. The createprocess system API call is 
used by an existing process, such as process A 1 06, to create a new process, such as 
process B 1 08. One of the parameters to the system API call is the name of the stored 
executable program file 136 (FIG. 1) containing the processor instructions forthe new 
process. When the createprocess system API call from process A 106 is intercepted by 
system call trap 300, and in accordance with an aspect of the invention, process B 
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attributes 202 and process B attribute extension 302 are created. In accordance with the 
described execution environment, access token B 215 (as modified by the 
execution environment attached to the stored program file specified in the createprocess 
system API call) is created using the execution environment attributes 225. The 
execution environment attributes 225 can be any attributes associated with a program 
file, and are provided to access token B 215 via connection 220. Similarly, the access 
token B extension 305, corresponding to process B attribute extension 302, also includes 
the execution environment attributes 225 supplied via connection 220. hi this manner, 
the execution environment dictates to the kernel 127 the attributes assigned to process B 
1 08. In this manner, the security attributes associated with a process can be altered or 
replaced based on the execution environment attributes attached to the stored program 
file represented by block 225. 

FIGS. 6A and 6B are flow charts collectively illustrating the preferred operation of 
the system. The flow charts of FIGS. 6A and 6B show the architecture, functionality, and 
operation of a possible implementation of the secure execution environment software. In 
this regard, each block represents a module, segment, or 
portion of code, which comprises one or more executable instructions for 
implementing the specified logical function(s). It should also be noted that in some 
alternative implementations, the functions noted in the blocks may occur out of the 
order noted in FIG. 6A. For example, two blocks shown in succession in FIG. 6 A 
may in fact be executed substantially concurrently or the blocks may sometimes be 
executed in the reverse order, depending upon the functionality involved, as will be 
further clarified below. 

With reference now to FIG. 6A, in block 401, the computing device 101 is 
"booted up" and runs initial system processes. In block 402, the system call trap 300 
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monitors all system API calls over API 120 (FIG. 5). In block 404 it is determined 
whether a system API call intercepted by system call trap 300 is a "create process- 
system API call. If the system call detected by system call trap 300 is not a 
createprocess system API call, then the process returns to block 402 and system call trap 
300 continues to monitor API 120 for system API calls. 

If, in block 404, system call trap 300 detects a "createprocess" system API call, 
then in block 406 it is determined whether the stored program file specified in the 
intercepted createprocess system API call has an execution environment (225 of FIG. 5) 
attached. If the file does not have an execution environment attached then the process 
returns to block 402. If the files does have an attached execution environment, such as 
that illustrated in FIG. 5, then in block 407 it is determined whether the execution 
environment is .narked inheritable, via the inheritable flag mentioned above. If the 
execution environment is not marked inheritable, then in block 408 a copy of the current 
security attributes are saved for a subsequent "createprocess'" system API call. These 
saved security attributes. are stored in the access token B extension 305 of FIG. 5. 

If it is determined in block 407 that the detected system API call includes an 
inheritable execution environment, then in block 409, and with reference now to FIG. 
6B, it is determined whether the current attribute set is marked inheritable. If the current 
attribute set is not marked inheritable, then in block 41 1 the detected execution 
environment attributes are merged with the previously saved set (block 408) to form a 
new set of attributes. 

If in block 409, the current attribute set is marked inheritable, then in block 4 1 2 
the execution environment attributes are merged with the current attributes to form a 
new set. In block 4 14, a new process (e.g., process B 1 08 of FIG. 5) ,s created using the 
newly created security attribute set (225 of FIG. 5). 
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In the preferred embodiment, the execution environment allows 
a computer system operator, or administrator, enhanced control over the operation of 
the computer system. By attaching execution environment attribute sets to various 
executable program files, the administrator can easily control the credentials and 
capabilities under which the various programs operate, irrespective of the identity of the 
invoking user, hi particular, the administrator can utilize the invention to achieve one of 
two primary effects, depending on whether the inheritable flag is enabled. The first, 
obtained when the inheritable flag is not enabled, allows the administrator to alter the 
security attributes or capabilities of a single program without conferring any additional 
capability on any subsequent process it may spawn. The second effect, obtained when 
the inheritable flag is enabled, allows the administrator to establish or re-establish the 
security attributes or capabilities of an entire process hierarchy (i.e. the executed process 
together with any processes it spawns and any process spawned by any descendant). 

In a typical configuration, only a few key program files may have an execution 
environment assigned. Other processes would simply inherit attributes from their 
predecessors. For example, the administrator can use these different attributes to 
effectively isolate a program (e.g., program 109 of FIG. 1) by performing the follow 
steps. Create a new "user" account for the program. Create an execution environment 
(e.g., execution environment 225 of FIGS. 1 and 5) associated with the program, 
assigning to it the newly created User ID, with that same User ID as the default owner 
of newly created files. Any files needed by that program would have their access 
control lists (138 of FIG. 1) modified so that only that user can access them. All other 
files 136 (i.e., those not needed by the application) might have their access control 
lists set to prohibit access by that user. 
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It will be apparent to those skilled in the art that many modifications and variations 
may be made to the preferred embodiments of the present invention, as set forth above, 
without departing substantially from the principles disclosed herein. For example, the 
system can be used to create a secure execution environment for any computer program 
that uses attributes to define access and security parameters. All such modifications and 
variations are intended to be included herein within the scope of the claims. 

The disclosures in United States patent application No. 09/517,366, from which 
this application claims priority, and in the abstract accompanying this application are 
incorporated herein by reference. 
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CLAIMS 



1 . A system for establishing a secure execution environment for a software process 
executed by a program operating on a computer, comprising: 

a software process operating on a computer, said software process including a 
plurality of attributes; 

an operating system kernel in communication with said software process and in 
communication with an executable file to be accessed by said software process; and 

a system call trap associated with said operating system kernel, said system call 
trap being configured to assign a selected plurality of said attributes to said software 
process, said selected plurality of attributes being stored in association with said 
executable file. 

2. A system as in claim 1 , wherein said system call trap comprises: 
a process attribute extension; and 

an access token extension associated with said process attribute extension, said 
access token extension including said selected plurality of attributes. 

3. A system as in claim 1 or 2, wherein said selected plurality of attributes are 
contained in a database associated with said executable file. 

4. A system as in claim 1 or 2, wherein said selected plurality of attributes are chosen 
from the group consisting of user ID, group IDs and privileges. 

5. A system as in any preceding claim, wherein said execution environment isolates 
said software process from any other software process operating on said computer. 

6. A method of establishing a secure execution environment for a software process 
executed by a program operating on a computer, the method comprising the steps of: 

operating a software process on a computer, said software process including a 
plurality of attributes; 

executing an operating system kernel in communication with said software process, 
said operating system kernel in communication with an executable file to be accessed by 
said software process; and 
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assigning a selected plurality of said attributes to said software process, said 
selected plurality of attributes being stored in association with said executable file. 

7. A method as in claim 6, comprising the steps of: 
executing a process attribute extension; and 

executing an access token extension associated with said process attribute 
extension, said access token extension including said selected plurality of attributes. 

8. A method as in claim 6 or 7, wherein said selected plurality of attributes are 
contained in a database associated with said executable file. 

9. A method as in claim 6 or 7, wherein said selected plurality of attributes are 
chosen from the group consisting of user ID, group IDs and privileges. 

10. A method as in any one of claims 6 to 9, wherein said execution environment 
isolates said software process from any other software process operating on said computer. 

11. A system for establishing a secure execution environment for a software process 
substantially as hereinbefore described with reference to and as illustrated in the 
accompanying drawings. 

12. A method of establishing a secure execution environment for a software process 
substantially as hereinbefore described with reference to and as illustrated in the 
accompanying drawings. 
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